1. Field of the Invention
The present invention generally relates to a software design and development system for software testing. In particular, the present invention relates to a method and system for generating an optimized suite of test cases.
2. Description of the Related Art
When testing software systems, developing an effective and efficient test suite is a complex problem. A suite is effective if it thoroughly covers combinations of inputs, actions, and sequences of actions, enhancing the likeliness it will encounter defects in the software. A suite is efficient if it provides such coverage without unnecessary redundancy, which would unduly increase the testing effort.
Typically, when a tester tests a software system, the tester or test program takes an action against the system being tested and receives a result or response back from the tested system. The received result is compared against an expected result to determine if the system is working correctly. The tester then takes another action gets some other response and the tester performs a series of these action/response sequences. Sometime, the order of this sequence depends upon the responses which were received earlier in the sequence. The number of ways that the sequences may be combined is very large.
Also, for most actions where the tester stimulates the system in a step in the sequence, the number of potential inputs to that particular action is very large. So, there are at least two places where one runs into a large number of combinations which leads to a combinatorial explosion problem.
Test generation is an area in which significant research has been performed. Most of the prior research has focused on finite-state based testing, which uses a finite state representation of the system being tested to generate sequences of actions which may be used to test the system. The actions are determined by seeking to cover states, transitions between states, or some combination thereof. These methods suffer from problems dealing with combinatorial explosion. Also, these techniques are very hard to optimize from an internal state-based viewpoint.
Briand and Labiche explore system testing using Unified Modeling Language (UML) use cases (see L. Briand and Y. Labiche, A UML-based approach to system testing. Technical Report SCE-01-01, Carleton University, June 2001). However, their published work to date focuses on use case dependency sequences, requirements derivation from sequence diagrams, and variant sequence derivation. That work does not address optimization or test generation problems.
Other applications of UML to testing include developing a UML-based representation for automating test generation for distributed systems, protocol conformance testing (L. Du Bousquet, H. Martin, and J.-M. Jezequel, Conformance testing from UML specifications experience report. In Proceedings of the Workshop on Practical UML-based Rigorous Development Methods, Toronto, CA, 2001, 43–55), component testing (S. Pickin, C. Jard, T. Heuillard, J.-M. Jezequel, and P. Desfray. A UML-integrated test description language for component testing. In Proceedings of the Workshop on Practical UML-based Rigorous Development Methods, Toronto, CA, 2001, 208–223), and the development of a standard representation in UML for describing conformance test cases (UML testing profile: Request for proposal, 2001, OMG document: AD/01-07-08. This work also does not address optimization of generation problems.
An additional difficulty in finite-state based test generation is model representation. Because the coverage focus of the test generation method is on system state, models often require a very high degree of detail and can be difficult to construct. The method described herein is based on the possible actions a system provides. This yields a much simpler model than many finite state-based methods.